Kalman filtering (KF) has good potential in fast rotation of state of polarization (RSOP) tracking. Different measurement equations cause the diverse RSOP tracking performances. We compare the conventional KF (CKF) and the modified KF (MKF), which have different measurement equations. Semi-theoretical analysis indicates the lower conditional variances of measurement residuals and process noise of MKF. Compared with CKF, the MKF has >3dB optical signal-to-noise ratio (OSNR) improvement at the 10 MHz scrambling rate in simulation. For MKF, more significant tracking speed improvement exists for lower OSNR. MKF can be smoothly combined with an adaptive algorithm, which outperforms adaptive CKF throughout the simulations.

We propose the modified Kalman filter (MKF) using the received signal for observation and constructing an inverse process of the conventional Kalman filter (CKF) for polarization de-multiplexing in coherent optical (CO) orthogonal frequency-division multiplexing (OFDM) transmissions. The MKF can avoid the convergence error problem in CKF without matrix inverse operation and has a faster converging speed and a much larger tolerance to the process and measurement noise covariance, about two orders of magnitude more than those of CKF. We experimentally demonstrate the 12 Gbaud OFDM signal transmission over 480 km standard single-mode fiber. The performance of MKF and CKF outperforms pilot-aided polarization de-multiplexing with better accuracy and nonlinearity tolerance.